fix(#999): provide TARGET_REPO_DIR to validation loop for Signed-off-by check - #1050
Conversation
|
🤖 Finished Review · ✅ Success · Started 5:47 PM UTC · Completed 5:58 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.32 |
|
Risk Assessment: moderate (2/5) DetailsTiny additive config change (3 lines across 3 files) by a known bot, exactly matching issue scope. Protected-path flag elevates T1 slightly (1.5). T2 is elevated (3.43) because all three files are high-churn with many fix-related commits and multiple distinct authors in the last 90 days, but the change itself is minimal and easily revertible. T3 is low (1.17) with clear issue alignment. Anchored to prior score of 2. Overall moderate risk. Previous runRisk Assessment: moderate (2/5) DetailsTiny additive config change (2 lines across 2 files) by a known bot, exactly matching issue scope. Protected-path flag elevates T1 slightly. T2 is elevated because harness/code.yaml and harness/fix.yaml are high-churn files with 11 commits in 30 days across 10 distinct authors and multiple recent fixes, but the change itself is minimal and easily revertible. Overall moderate risk. |
ReviewFindingsMedium
Previous runReviewFindingsMedium
Previous run (2)ReviewFindingsMedium
|
fullsend-ai-review
Bot
added
the
requires-manual-review
|
/ok-to-test |
|
/fs-fix The functional tests fail because Fix: in This mirrors production where REPO_DIR=${GITHUB_WORKSPACE}/target-repo. In the eval harness, TARGET_DIR is already set to ${EVAL_GH_WORKSPACE}/target-repo, so the value is equivalent. |
|
🤖 Finished Fix · ✅ Success · Started 8:30 PM UTC · Completed 8:36 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.29 |
Add `emit_env "REPO_DIR" "${TARGET_DIR}"` to the code|fix case block
so that harness env validation accepts the TARGET_REPO_DIR: "${REPO_DIR}"
references added in harness/code.yaml and harness/fix.yaml. Mirrors
production where REPO_DIR=${GITHUB_WORKSPACE}/target-repo.
Addresses review feedback on #1050
🔧 Fix agent — iteration 1 (human-triggered)Added emit_env REPO_DIR to the code|fix case block in eval/scripts/run-fullsend.sh, fixing fullsend env validation failure caused by unset REPO_DIR referenced by TARGET_REPO_DIR in harness YAML files. Fixed (1):
Tests: passedNext steps:
|
|
🤖 Finished Review · ✅ Success · Started 8:38 PM UTC · Completed 8:53 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.37 |
fullsend-ai-review
Bot
added
the
requires-manual-review
…by check
The validate-code-output.sh script guards its Part 2 (Signed-off-by
trailer check and pre-commit gate) behind a TARGET_REPO_DIR
conditional. When TARGET_REPO_DIR is not set, Part 2 is silently
skipped and only schema validation runs. Neither the fix nor code
harness provided TARGET_REPO_DIR in their env.runner blocks, so the
validation loop never caught Signed-off-by trailers — they passed
in-sandbox validation and were only caught by the post-script hard
gate, forcing unnecessary human re-dispatch.
Add TARGET_REPO_DIR: "${REPO_DIR}" to env.runner in both
harness/fix.yaml and harness/code.yaml so the validation loop script
receives the path to the target repo and runs the full Part 2 check
(Signed-off-by + pre-commit gate) during in-sandbox retry iterations.
Closes #999
Add `emit_env "REPO_DIR" "${TARGET_DIR}"` to the code|fix case block
so that harness env validation accepts the TARGET_REPO_DIR: "${REPO_DIR}"
references added in harness/code.yaml and harness/fix.yaml. Mirrors
production where REPO_DIR=${GITHUB_WORKSPACE}/target-repo.
Addresses review feedback on #1050
890b840 to
580410e
Compare
|
🤖 Finished Review · ✅ Success · Started 12:45 AM UTC · Completed 12:59 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.58 |
|
🤖 Finished Retro · ✅ Success · Started 1:19 AM UTC · Completed 1:29 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.21 |
Retro: PR #1050 — provide TARGET_REPO_DIR to validation loopTimelineIssue #999 (created by the retro agent) identified that the fix/code agent validation loop silently skipped Part 2 of What went well
What could be improvedCode agent missed the eval script dependency. The code agent read Evidence for existing open issues
Proposals filed
|
… REPO_DIR Production workflows set TARGET_REPO_DIR via setup-agent-env.sh (stripping the FIX_/CODE_ prefix). The harness YAML files introduced in fullsend-ai#1050 referenced ${REPO_DIR}, which is never set in production — only in the eval script. Change the harness to pass-through ${TARGET_REPO_DIR} and update the eval script to emit TARGET_REPO_DIR to match. Closes fullsend-ai#1062 Signed-off-by: Greg Allen <gallen@redhat.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 participant
Summary
TARGET_REPO_DIR: "${REPO_DIR}"toenv.runnerinharness/fix.yamlandharness/code.yamlvalidate-code-output.shPart 2 checks (Signed-off-by trailer detection and pre-commit gate) during the in-sandbox validation loop, so agent-fixable errors trigger a retry iteration instead of reaching the post-script hard gateProblem
The
validate-code-output.shscript guards Part 2 behind aTARGET_REPO_DIRconditional — when the variable is unset, the script soft-passes after schema validation and never checks for Signed-off-by trailers. Neither harness providedTARGET_REPO_DIRinenv.runner, so the validation loop only ran schema validation (Part 1). Signed-off-by trailers passed in-sandbox validation and were caught only by the post-script, which terminally rejected the push and required human re-dispatch.Testing
validate-code-output-test.shtests pass (includingsignoff-consumes-iterationwhich exercises the exact Signed-off-by detection path)TARGET_REPO_DIRis present in both harness configslint-agent-docshook passesCloses #999
Post-script verification
agent/999-fix-validation-target-repo-dir)2fbb7da338097ad4c3d659820444c7523a0f1e63..HEAD)